Model-independent comparison of simulation output

Computational models of complex systems are usually elaborate andsensitive to implementation details, characteristics which often affect theirverification and validation. Model replication is a possible solution to thisissue. It avoids biases associated with the language or toolkit used todevelop the original model, not only promoting its verification and val-idation, but also fostering the credibility of the underlying conceptualmodel. However, different model implementations must be comparedto assess their equivalence. The problem is, given two or more imple-mentations of a stochastic model, how to prove that they display simi-lar behavior? In this paper, we present a model comparison technique,which uses principal component analysis to convert simulation outputinto a set of linearly uncorrelated statistical measures, analyzable in aconsistent, model-independent fashion. It is appropriate for ascertainingdistributional equivalence of a model replication with its original imple-mentation. Besides model-independence, this technique has three otherdesirable properties: a) it automatically selects output features that bestexplain implementation differences; b) it does not depend on the distribu-tional properties of simulation output; and, c) it simplifies the modelers’work, as it can be used directly on simulation outputs. The proposedtechnique is shown to produce similar results to the manual or empiri-cal selection of output features when applied to a well-studied referencemodel. Keywords— Model Alignment; Docking; PCA; Model Replication; Simu-lation Output Analysis